Definition¶
A domain $\mathcal{D}$ consists of a feature space $\mathcal{X}$ and marginal probability distribution $P(X)$ over the feature space, where $X = x_1, ..., x_n \in \mathcal{X}$.
Given a domain, $\mathcal{D} = \left\{\mathcal{X}, P(X)\right\}$, a task $\mathcal{T}$ consists of a label space $\mathcal{Y}$ and a conditional probability distribution $P(Y|X)$ learned from the training data.
Given a source domain $\mathcal{D}_S$, a corresponding source task $\mathcal{T}_S$, as well as a target domain $\mathcal{D}_T$ and a target task $\mathcal{T}_T$, the objective of transfer learning is to learn the target conditional probability distribution $P(Y_T|X_T)$ in $\mathcal{D_T}$ with information gained from $\mathcal{D}_S$ and $\mathcal{D}_T$ where $\mathcal{D}_S \neq \mathcal{D}_T$ or $\mathcal{T}_S \neq \mathcal{T}_T$.
Settings and Approaches¶
Early approaches to transfer learning defined three different settings within transfer learning:
- Inductive - same domains, different tasks
- Unsupervised - different domains, different tasks
- Transductive - different domains, same tasks
Along with these settings are four different approaches that are be used in these settings:
- Instance-transfer - re-weight some labeled data in source domain for use in target domain
- Feature-representation transfer - find a feature representation that reduces the difference between the source and target domains
- Parameter-transfer - Discover shared parameters between domains
- Relation-knowledge transfer - Build mapping of relation knowledge between domains
The table below shows how these approaches can be used in different settings:
(Sinno Jialin Pan and Qiang Yang, A Survey on Transfer Learning)
Applications and Examples¶
Spam Filtering¶
A transfer learning example in which the domains are the same, but tasks are different, would be a classifier to detect spam in email. A model could be trained on emails from multiple users. A new email user could then use this model to filter their own messages.
Domain Adaptation¶
An example in which domains differ but tasks are the same would be a classifier to detect bicycles, but the target domain has very little data (bikes in the wild), and the source domain has a lot of data (bikes in the lab).
(Sun, B., Feng, J., & Saenko, K. (2016). Return of Frustratingly Easy Domain Adaptation)
Pre-trained CNN features from large CNNs¶
Pre-trained CNN features are often used as a form of transfer learning in a variety of tasks such as image classification, object detection, style transfer and generative models. Large CNNs that have been trained on ImageNet learn the underlying structure of images, which is knowledge that can be transferred. A lot of recent research in computer vision and deep learning tend to use VGG16 or VGG19 as a base to extract features from the target domain.
Object Detection¶
State of the art object detection is done by YOLO, which pretrains with ImageNet, to learn knowledge from a classification task before training the model for object detection.
Style Transfer¶
Johnson et al's real time style transfer use the extracted features from VGG19 in the loss function used to train the style transformation network. The features can be used as statistics measuring the style and content of images.
Super Resolution¶
Another application of using VGG19 in a loss function is with super resolution. The loss is taken as the difference between the extracted features of the output image and the actual high resolution image.
Sentiment Analysis¶
Applications in NLP include learning from a large set of labeled reviews, and transferring to a model to analyze reviews for a new product.
